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Abstract. We prove that if a rectangular r x n matrix with uniformly small 
entries and approximately orthogonal rows is applied to the independent stan- 
dardized random variables with uniformly bounded third moments, then the 
empirical CDF of the resulting partial sums converges to the normal CDF 
with probability one. This implies almost sure convergence of empirical peri- 
odograms, almost sure convergence of spectra of circulant and reverse circulant 
matrices, and almost sure convergence of the CDF's generated from indepen- 
dent random variables by independent random orthogonal matrices. 

For special trigonometric matrices, the speed of the almost sure convergence 
is described by the normal approximation and by the large deviation principle. 



1. Results 



The study of spectra of circulant matrices lead Massey, Miller and Shinsheimer 
[T^ to an almost sure Central Limit Theorem (CLT) which takes a different form 
than the celebrated almost sure CLT discovered by Brosamler |Hj and Schatte JH| ■ 
Our main result extends |14l Theorem 5.1] in several ways: we allow more general 
weights, we do not assume identical distributions, we do not assume higher moments 
than three, and we prove multivariate convergence. We consider weighted sums 
of independent random variables with the weights that come from matrices with 
"almost orthogonal" rows and uniformly small entries. Namely, let U 



(n) 



be a family of real r„ x n matrices, n > 1. 
C,S > which do not depend on n = 1,2,.. 



(1.1) 



(1.2) 



max 

l<j<n,l</£<r„ 



max 

l<fci,fc2<r„ 



'Si 



We assume that for some constants 
, we have 

C 



< 



< 



(log(l + r„))i+^ 



C 



(log(l + r„))i+^ 



For application to periodograms, we will also need to consider pairs 
such matrices , and then we will assume that in addition we have 



(1.3) 



max 

l<fci,fc2<r„ 



< 



c 



(log(l + r„)) 



1+.5- 
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An example of a sequence of such pairs U*^"-' , V^"^ of matrices with r„ < [^^^^J is 

(1^4) ^ vf (^) • <1 - Vf (^) . 1 s s ». 1 s * s 

Then holds trivially, while p.2|) . and p.3|l follow from the fact that 2r„ < n 
and the following well known trigonometric identities hold for I < ki < k2 < n: 

E-^(^^j-«(^^j - [n/2 iffc, + fc,=n ' 

n fl^ V- • /^27rfcijA . f 2TTk2j\ / if fci + A:2 ^ n 

(1-6) - [-n/2 itk,+k, = n ' 



(1.7) ^cos — — sm — — = 0. 



j=i 

In addition, for 1 < fc < n 



(1.8) 



Denote by ^{x) the standard normal cumulative distribution function; N{m, cr^) 
denotes the normal law with mean m and variance ct^; X„ X denotes weak 
convergence of laws. All random variables are assumed to be defined on a common 
probability space (fi, JF, P). 

To avoid cumbersome notation, we state the theorem for the univariate and 
bivariate cases only. The m-variate extension requires introducing m sequences of 
matrices that satisfy conditions l|l.l|l and (|1.2(l . with each pair from the n-th level 
satisfying condition (|1.,S|I : the proof requires only minor changes. 

Theorem 1.1 (Almost sure CLT). Suppose Xi,X2, ■ ■ ■ are real independent ran- 
dom variables such that E{Xj) — 0, E{Xj) — 1, and swpj^ E\Xk\^ < oo. Fix 
r„ / oo. 

(i) Forn=\,2,..., let Ij'-''^ be an rn X n matrix such that conditions (|l.l|l . 
(|1.2|1 hold with some C,S independent of n. Define 

n 

(1.9) 5„,fc=^45^J' fc = l,2,...,r„. 



Then with probability 1, 



(1.10) lim sup 



k=l 



0. 



(ii) For n = 1,2,..., let V(") be an rn X n matrix such that conditions (jl.l|l . 
(11.2(1 hold for v^^j in place ofu^j^j , and such that the pairs (U*^"-' , V^"' )n=i2 
satisfy condition (|1.3|l . Define 

n 

(1.11) = ""S^i' fc - 1, 2, . . . , r„. 
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Then there is a measurable set flo C of probability 1 such that for all 

T-n. 

(1.12) limr-iVl{ 

fc=l 

We also have the companion weak limit theorem and the large deviation principle 
for the univariate case with trigonometric coefficients given by H1.4|l under the 
restrictions on the rate of growth of r„. The most interesting case, r„ = [(n — 
l)/2], which corresponds to the spectral measures of random circulant matrices, is 
unfortunately not covered by our result; LDP for spectra of other random matrices 
are in (9| Chapter 5] . 

Proposition 1.2. Suppose Xi,X2, ■ ■ ■ are real independent random variables such 
that E{Xj) = 0, E(X'^) = 1, and for some constant t > 0, 

(1.13) sup^(|Xfc|3exp(|Xfe|/r)) < r. 

fe 

LetU(") be given by H1.4|l . and suppose that r„ — > oo is such that 
(1.14) 

Then for x Cz R, 

(1.15) — ^ (ls„ ,<x - ^{x)) N (0, $(.t)(1 - $(x))) asn-^^, 

v*"" k=l 

Condition (|1.13l) holds for the i.i.d. sequences with r that depends on the law 
of Xi when E{exp{S\X\)) < oo for some 6 > 0. We remark that (|1.15(l holds true 
with assumptions H1.13|l and H1.14|l replaced by the assumption that there is p > 
such that 

supE(\Xk\^+P) < oo, and r^n-f/^^+p) ^ q. 
fc 

This follows from our proof, after substituting |17[ Section 5, Corollary 5] for Lemma 

The large deviation principle (LDP) was motivated by ^IS?; Theorem 1], which 
gives the LDP from the Brosamler-Schatte almost sure CLT. To formulate the result 
we need more notation. Let A4i(IR) denote the Polish space of probability measures 
on the Borel sets of E with the topology of weak convergence. For Sn,k given by 
1)1.9(1 . consider the empirical measures 



(1.16) f,,^^lY,5 



■ " fc=i 

The rate function / : A1i(R) [0,oo] in our LDP is the relative entropy of with 
respect to the standard normal law, i.e. if ^(a;) denotes the normal density and 
ly € A^i(K), then /(z/) = / In ^^^f{x)dx if v has the density / with respect to the 
Lebesgue measure and the expression is integrable, and I^iy) — oo otherwise. It is 
well known that the level sets /~^[0,a] are compact for a < oo. 

The conclusion of the next result is the LDP of the empirical measures with 
speed Tn and the rate function /(•). 
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Proposition 1.3. Suppose matrices U'^"-' and random variables Xi,X2, ■ ■ ■ are as 
in Proposition If rn is such that 

> ana > 0, 

n r„ 

then for all open sets G and closed sets F in Ali(]R), 

liminf — lnPr(/^„ e G) > - inf I{i>) 

and 

limsup — lnPr(^„ & F) < — inf 
We postpone the proofs to sectional and we first give some applications. 



1.1. Application to periodograms. The periodogram of a sequence {Xj) is 

1 



/„(A) = 



/n 



3) 
2 



The empirical distribution of the periodogram is the (random) CDF 

1 71—1 
Fn(x) ■■= 2^ l{/„(277fe/n)<K}, rn = [ J. 



^" fc=l 



Theorem 11.11 strengthens the conclusion of [111 Proposition 4.1] to almost sure 
convergence at the expense of the assumption that third moments are finite. 

Corollary 1.4. If Xi,X2, ■■ ■ are independent with E{Xj) = 0, E{Xj) = 1, 
EQXjl^) < M < oo, then sup^>o \Fnix) — (1 — e~^)| with probability 1. 

Proof. Following |11[ (2.1)], we write F„(.t) = /i„((— oo,x]) as the CDF of the 
empirical measure 



1 



" fc=l 



where Sn.k and T„^fc are defined by 1)1. 9|l and (|1.11() with matrices U'^"\ V*^") given 
by H1.4|l . The result follows from Theorem 11.11 if ft, : ^ is a continuous 
mapping of Polish spaces and discrete measures ^ X]fc=i converge weakly to 
some probability measure u on the Borel sigma-field of i?, then the discrete measures 
— Y^^=i^h(xk) converge weakly to the probability measure v o h~^, see e.g. |31 
Theorem 29.2] . We apply this to ft : ^ R given hy h{x , y) = x'^ + and to z/ on 
Borel sets of R^, which is the product of the standard normal laws. Then i/oh^^ is 
the Chi-Squared law with 2 degrees of freedom, i.e. it is the standard exponential 
law with the CDF given by 1 — e~^ for a; > 0. Since the limit (1 — e~^)_|_ is a 
continuous CDF, it is well known, see jSl Exercise 14.8], that the convergence is 
uniform with respect to x. □ 
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1.2. Application to symmetric circulant and reverse circulant matrices. 

Corollary 1.5. The weak convergence in and (4j Theorem 5] holds with proba- 
bility one. 

Proof. Ref. "S" and 4, Theorem 5] analyze the asymptotic spectrum of the n x n 

symmetric random matrices with the typical eigenvalues of the form ± '\J S'^ j, + f^, 

where fc = 1, 2, . . . , [^\ , and U^"), V(") defined by (IT3J), see Lemma 1]. Omit- 
ting at most two eigenvalues does not change the convergence of the spectral mea- 
sure, so theorem 11.11 implies that the convergence holds with probability one by the 
argument similar to the proof of Corollarv ll.4l □ 

Suppose A„ is a symmetric random circulant matrix formed from the indepen- 
dent random variables by taking as the first row [A„]ij- — Xj, j — 1, 2, . . . , [(n -|- 
l)/2] and [A„]i.j ~ [A„]i^„_j for other j. The next corollary strengthens jSj Re- 
mark 2] to almost sure convergence, and relaxes the integr ability and i.i.d assump- 
tion in Theorem 1.5]. To justify the later claim, we note that a "palindromic 
matrix" analyzed in |14| differs from A„ by the last row and column only; thus 
their ranks differ by at most one, and asymptotically "palindromic matrices" and 
random circulant matrices have the same spectrum, see JT^, Lemma 2.2]. 

Corollary 1.6. If Xi, X2, . ■ . are independent with uniformly bounded third mo- 
ments, with common mean E(Xj) — m and common variance Var{Xj) — > 0, 
then the spectrum of ^-i^A„ converges weakly with probability 1 to the standard 
normal law. 

Proof. Subtracting the rank 1 matrix does not change the asymptotic of the spec- 
trum, thus without loss of generality we may assume m = 0; rescaling the variables 
by (7 > we can assume E{X'j) = 1. With the exception of at most two eigenval- 
ues, the remaining eigenvalues of A„/-y/n are of multiplicity two and are given by 
(|1.9|l with the trigonometric matrix U^"-' given by (|1.4|) . see Remark 2]. Thus 
the weak convergence with probability one of the spectral law of A^/ -Jn to N [0,1) 
follows from Theorem ll.il □ 

1.3. Application to random orthogonal matrices. A well known result of 
Poincarc says that if U(") is a random orthogonal matrix uniformly distributed on 
0{n) and Xn G is a sequence of vectors of norm ^/n then the first k coordinates 
of U'^"^a;„ are asymptotically normal and independent, see e.g. |2| Exercise 29.9]. 

Corollary 1.7. Let Xi,X2,... be independent, E{Xj) = 0, E{Xj) = 1 and 
snpj E{\Xj\'^) < 00. Let U*-"^ be a random orthogonal matrix uniformly distributed 
on 0{n) and independent of (Xj). Define Sn,k by (|1.9|l . Then with probability one 

(1.17) sup 

Proof. This result has a direct elementary proof, which we learned from Jack Sil- 
verstein. This proof shows that the result holds true also for i.i.d. random variables 
with finite second moments. Here we derive it as a corollary to Theorem ll.il 

Orthogonal matrices satisfy H1.2(l with r„ = n. By ^| Theorem 1], Hl.l|l holds 
with probability 1. Therefore, redefining U^"^ and (Xj) on the product probability 



fc=i 



/27r 



-^"/^du 



as n — !■ 00 . 
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space flu X fix, by fTD| Theorem 1], there is a subset il'jj of probabihty 1 such 
that for each uji E fl'jj by Theorem 11.11 there is a measurable subset flx.uji C fix 
of probabihty one such that (|1.17|) holds. By Fubini's Theorem, the set of all pairs 
{uji,uj2) for which H1.17|l holds has probability one. □ 



2. Proofs 

2.1. Proof of Theorem II. IL The proof of consists of several lemmas. 

Lemma 2.1 f jl5l Theorem 2.6]). It is enough to verify almost sure convergence of 
characteristic functions. 

To use Lemma l2.1l fix real s, t and consider the (random) characteristic function 
$„(s,t,a;) = r^^ ^exp(isS'„_fc(w) + itTn,k[<^)). 

k=l 

Lemma 2.2. There is C = C{t) that does not depend on n > 1 such that for all 
large enough n > N{t) we have 

(2.1) i?|$„(s, t, - e-^'/2-tV2|2 < c/(log(l + r„))i+*\ 

Proof. The left hand side of H2.1|l is 



"2—6 



fcl,/C2 = l 

Denote ipj{t) = E{e'*^^). Then the left hand side of ijTTJ can be bounded by 

n n 



< 2/r„ + max 



2 max max 

± l<fe<r„ 



Y[^,i±s4:j±tvS)-e-^^'+^y^ 



We will show how to bound the middle term, as the last one is handled similarly. 
Trivially, 



(2.2) max 
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where 



n 

(2.3) A. = max | J] (^(4:!, " + ti<], ~ -tlS) 

n , 



(2.4) 



max 



exp 



By for large enough n we have 



(2.5) 



< max 

j,ki,k2 



(») 



< 1. 



We now use the well known bound, see e.g. (27.13)], which bounds the first 
term by 



A„<cmax^|.(4:).-4:)o+^(4:!.-4:!.) 



< 



(log(l + r„))i+* 



(log(l + r„)) 



1+5 ■ 



The second term is bounded using p.2|) and as follows. We first note that 

ki ^ k2 implies 



where 

|i?(n, fci, fe, s, i)| < 



E((-a)^+(4 



■2 j' ' 



- 2 



E((4:i)^ + (4S)^ 



+ 2s" 



(") „(") 



E(n 

J = l 



+ 2i^ 



E(n) (n) 
U, -V, ■ 



j-V^ (") (") (") (") I (") (") (") (") 

sty u) v, ■ — u) v, ■ + u; v) - — u, v, ■ 



< 



C(s,t) 



(log(l + r„)) 



1+5 ■ 



Since |e° — e''\ < \a — b\ max^g^^.h] e", and H2.5|l holds, therefore for large enough n 
this implies 

C{s,t) 



^" - (log(l + r„))i+^- 
To prove almost sure convergence we will use the following. 



□ 
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Lemma 2.3 ( jl2l Theorem 1]). Let Yi, Y2, . . . be uniformly hounded C-valued and 
possibly dependent random variables, r„ 00. Suppose Zn — T-X^lLi^fc- V 
E\Z„\'^ < C/(log(l + r„))i+* then Z„ with probability 1. 

Proof of Theorem TTl\ To prove part (ii), let Yfc = exp(zsS'„,fc+ztT„_fc)-e~''^/^^*^/^. 
Then by Lemma 12.21 the assumptions of Lemma 12.31 are satisfied. So Zn(j-^) — 
$„(s,i,[x)) — — + with probabihty one. By Lemma [2.11 this imphes 

The proof part (i) is similar, and essentially consists of taking i = in the above 
calculations; once we establish the weak convergence on a set Hq of probability 1, 
due to continuity of the convergence is uniform in x for every uj £ Qq, see |31 

Exercise 14.8]. 

□ 

2.2. Proof of Propositions 11.21 and 11.31 The proofs rely on strong approxi- 
mation of the partial sum processes indexed by the Lipschitz functions fk{x) = 
cos(27rfca;), compare |H1 Theorems 2.1, 2.2]. We derive suitable approximation di- 
rectly from the following result. 

Lemma 2.4 (Sakhanenko [TBI Theorem 1]). Suppose Xi,X2,-.- satisfy the as- 
sumptions of Proposition IJ.jjl Then there is a constant c such that for every n 
one can realize Xi, X2, ■ ■ ■ , Xn on a probability on which there are i.i.d. 7V(0, 1) 
random variables Xi,X2, ■ ■ ■ ,Xn such that the partial sums Sj — -^i '^^'^ 

= Y.i=i satisfy 

(2.6) £;(exp(^^maxJ5,-S^,|))<l + ^. 

(Recall that t is defined in H1.13|l . ) 

We use this lemma as follows. For every n, we redefine Xi,X2, ■ ■ ■ ,Xn onto a 
new probability space {Qn,J-n,Pn) on which we have the i.i.d. standard normal 
r.v. Xi,X2, . . . , Xn which satisfy (|2.6() . We then define {Sn.k ■ fc = 1,2,..., r„} by 
(|1.9|l . and we also define 

(2.7) ^ = Vx,cos(^), fc = l,2,...,r„. 

(Clearly we should have used the triangular array notation {Xj n)j instead of {Xj)j; 
we can safely omit the subscript n here, since its re-appearance in the partial sums 
S„^k keeps track of n anyway.) 

The assumptions and the conclusions of ProDOsitions ll.2l and ll.3l are not affected 
by such a change. 

We note that from the trigonometric identities H1.5I1.8|I , it follows that for fixed 
n random variables Sn,i, Sn,2, ■ ■ ■ , Sn,rr, are i.i.d standard normal. Therefore, if 
r„ — > 00 then for any 77 we have 
(2.8) 

^ E {^s„..<.+v/V^ " * ("^ + ) ^ ^ ^^"^^^^ " '^^''^^^ as n ^ 00. 

(This is just the normal approximation to the binomial random variables with the 
probabilities of success ^ (^x + -^=j ^i^)-) 
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Consider now the corresponding empirical measures 
(2-9) = 



k=l 



If r„ oo, then by Sanov's Theorem, see e.g. for all open sets G and closed 
sets F in 



and 



liminf — lnPr(^„ e G) > - inf 



limsup — lnPr(/x„ (z F) < — inf /(i^). 



The plan of proof is to deduce Propositions 11.21 and 1 1 . 31 from these two facts. 

By taking a product space, without loss of generality we assume that all random 
variables Sn,k,Sn,k, n- > 1,1 < k < r„, are defined on the common probability 
space. We never need joint distributions of these variables for different n, but such 
a choice simplifies the notation. 

Proof of Proposition M.HX Denote 



k=l 



and let Zn{x) denote the corresponding sum for the i.i.d. normal random variables 
(|2.7|l from Lemma Fix e > 0, and let e„ = ey^27r/r^. From the trivial bound 

|$(a;±e„) - $(x)| < e„/\/2^ 

we get 

(2.10) Zn{x - e.n) - ^/n^'^{x - £n) - e ~ Rn 

< Zn{x) - ^i<l>{x) < Zn{x + £„) - ^^{x + £„) + £ + i?„, 

where 

Rn = \/r^lA„, An = { max \Sn,k - Sn,k\ > En}. 
l<k<rn 

Since cos(-) is a Lipschitz function, 

(2.11) max IS'n^fe - S^„,fc| = max 



l<fe<r 



— 1/2 

< n ' max 

fc<r„ 



j=i \ / 



n-l 



Y^^Q Q^f /^27ri/c\ f 2Tr{j + l)k 
X:(5, - 5,) (^cos j cos 



+ n-^/^\Sn-Sn\ 



27rfc 



< max Y \S, - S,\ + n-^/^\S„ - 5„| 

k<r-- ^-^ •' n 



1 + 2^r„ ~ 
< — — max i), — Sj 

^ i<i<« 
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From Lemma [2.41 we see that for large enough n so that rnjyfn < c/t we have by 
Markov inequaUty 

Pr(A„) < Pr I max \Sj - Sj\> 



\<2<n ^r;r(l + 27rr„) 



cev27m 

r;7(l + 27rr„) / " y—^- y-' ■ {^JZx ' 



< exp I — _ - — ^ ^ ) E ( exp ( c/t rnax \Sj — Sj 



Thus Rn — > in probability. Since e > is arbitrary, Proposition 1 1 . 21 follows from 
ijTTUI by (E31). □ 

Our proof of Proposition 1 1 . is based on the following approximation lemma. 

Lemma 2.5 ([21 Theorem 4.9]). Suppose the families of real random variables 
Sn,k, Sn,k,n > 1, 1 < < o-f^ such that for every 9 > 

(2.12) limsupr-Mogi; l^exp |^^?^|5„,fc-5„,fc|^^ < 1. 

// the empirical measures (I2.9|l satisfy the LDP in Mi (M) with speed rn and a good 
rate function !{■), then the empirical measures ((rTBI) satisfy the LDP in Mi{R) 
with the same speed rn and the same rate function /(•). 

Proof of Provosition \l.'A Since the Large Deviation Principle for pt„ follows from 
Sanov's Theorem, to end the proof we only need to verify assumption H2.12|l of 
Lemma [2.51 Using (|2.11f) . we see that for large n there is C such that 

E ^exp(6'^ \Sn,k - Sn,k\)^ < E ^exp(6'r„ ^max \Sn,k - Sn,k\)j 

< E ( exp(Cr^n-^/2 max - Sj\ 

\ l<j<n 

Since r^n^^/^ — > 0, therefore for large enough n we can apply 1)2. 6|l . We get 
r-i logi? (exp(0|i \Sn,k - SnM)^ < ci^^iLtzVl) ^ q 



□ 
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